模式识别与人工智能
Sunday, Apr. 6, 2025 Home      About Journal      Editorial Board      Instructions      Ethics Statement      Contact Us                   中文
Pattern Recognition and Artificial Intelligence  2023, Vol. 36 Issue (11): 967-986    DOI: 10.16451/j.cnki.issn1003-6059.202311001
Current Issue| Next Issue| Archive| Adv Search |
Synesthesia of Machines Towards Intelligent Multi-modal Sensing-Communication Integration
CHENG Xiang1, ZHANG Haotian1, LI Sijiang1, HUANG Ziwei1, YANG Zonghui1, GAO Shijian2, BAI Lu3,4, ZHANG Jia'nan1, ZHENG Xinhu5, YANG Liuqing5,6,7
1. School of Electronics, Peking University, Beijing 100871;
2. Samsung Semiconductor, Samsung SoC Research and Deve-lopment Lab, San Diego, CA 92121, USA;
3. Shandong Research Institute of Industrial Technology, Jinan 250100;
4. Joint SDU-NTU Centre for Artificial Intelligence Research, Shangdong University, Jinan 250101;
5. Intelligent Transportation Thrust, The Hong Kong University of Science and Technology(Guangzhou), Guangzhou 511455;
6. Internet of Things Thrust, The Hong Kong University of Science and Technology(Guangzhou), Guangzhou 511455;
7. Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong 999077

Download: PDF (2844 KB)   HTML (1 KB) 
Export: BibTeX | EndNote (RIS)      
Abstract  Integrated sensing and communications(ISAC) technique is limited to the sharing of radar sensing and communications at the spectrum and hardware levels, and it fails to enhance the performance of communication and sensing in future emerging application scenarios. In scenarios involving massive multi-modal sensing and communication data, ISAC should evolve towards the incorporation of multi-modal sensing, specifically intelligent multi-modal sensing-communication integration. Inspired by human synesthesia, a paradigm for intelligent multi-modal sensing-communication integration, synesthesia of machines(SoM), is systematically established and discussed in this paper. Firstly, three typical operational modes of SoM , SoM-evoke, SoM-enhance and SoM-concert, are systematically summarized, and thus the purposes and methods of the mutual assistance and enhancement between communications and multi-modal sensing are given comprehensively. Then, the data foundation of SoM research, mixed multi-modal sensing and communication(M3SC) simulation dataset, and the theoretical foundation of SoM research, SoM mechanism, are also discussed. Finally, the current research status of SoM is reviewed and future research directions are prospected.
Key wordsSynesthesia of Machines(SoM)      Integrated Sensing and Communications      Multi-modal Sensing      Artificial Neural Networks      Synesthesia of Machine Mechanism     
Received: 10 October 2023     
ZTFLH: TN92  
Fund:National Key Research and Development Project of China(No.2020AAA0108101), National Natural Science Foun-dation of China(No.62125101,62341101,62001018,62301011,62373315,U23A20339,62371273), New Cornerstone Science Foundation through the XPLORER PRIZE, Natural Science Foundation of Shandong Province(No.ZR2023YQ058), Young Elite Scientists Sponsorship Program by CAST(No.2023QNRC001), Taishan Scholars Program, Guangzhou Municipal Science and Technology Project(No.2023A03J0011,2023A03J06831), and Guangdong Provincial Department of Education Major Research Project(No.2023ZDZX1037)
Corresponding Authors: CHENG Xiang, Ph.D., professor. His research interests in-clude data-driven intelligent network and net-worked intelligence.   
About author:: ZHANG Haotian, Ph.D. candidate. His re-search interests include multi-modal sensing-assisted transceiver design in communication systems.Li Sijiang, Ph.D. candidate. His research interests include cooperative localization, per-ception and intelligence of connected mobile agents.HUANG Ziwei, Ph.D. candidate. His re-search interests include complex high-mobility communication channel measurements and mo-deling.YANG Zonghui, Ph.D. candidate. His re-search interests include wireless communica-tions and integrated sensing and communi-cations.GAO Shijian, Ph.D., senior engineer. His research interests include wireless communica-tions and statistical signal processing.BAI Lu, Ph.D., professor. Her research interests include 6G wireless communication network channel measurements and modeling.ZHANG Jia'nan, Ph.D., assistant profe-ssor. His research interests include network theory and networked intelligence.ZHENG Xinhu, Ph.D., assistant professor. His research interests include multi-modal perception, multi-agent cooperative perception and networked intelligence.YANG Liuqing, Ph.D., professor. Her research interests include wireless communica-tion networks, multi-agent systems and inte-grated communication and sensing.
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
CHENG Xiang
ZHANG Haotian
LI Sijiang
HUANG Ziwei
YANG Zonghui
GAO Shijian
BAI Lu
ZHANG Jia'nan
ZHENG Xinhu
YANG Liuqing
Cite this article:   
CHENG Xiang,ZHANG Haotian,LI Sijiang等. Synesthesia of Machines Towards Intelligent Multi-modal Sensing-Communication Integration[J]. Pattern Recognition and Artificial Intelligence, 2023, 36(11): 967-986.
URL:  
http://manu46.magtech.com.cn/Jweb_prai/EN/10.16451/j.cnki.issn1003-6059.202311001      OR     http://manu46.magtech.com.cn/Jweb_prai/EN/Y2023/V36/I11/967
Copyright © 2010 Editorial Office of Pattern Recognition and Artificial Intelligence
Address: No.350 Shushanhu Road, Hefei, Anhui Province, P.R. China Tel: 0551-65591176 Fax:0551-65591176 Email: bjb@iim.ac.cn
Supported by Beijing Magtech  Email:support@magtech.com.cn